While tangible user interface has shown its power in naturally interacting with rigid or soft objects, users cannot conveniently use different types of granular materials as the interaction media. We introduce DipMe as a smart device to recognize the types of granular media in real time, which can be used to connect the granular materials in the physical world with various virtual content. Other than vision-based solutions, we propose a dip operation of our device and exploit the haptic signals to recognize different types of granular materials. With modern machine learning tools, we find the haptic signals from different granular media are distinguishable by DipMe. With the online granular object recognition, we build several tangible interactive applications, demonstrating the effects of DipMe in perceiving granular materials and its potential in developing a tangible user interface with granular objects as the new media.

This paper introduces Sketched Reality, an approach that combines AR sketching and actuated tangible user interfaces (TUI) for bidirectional sketching interaction. Bi-directional sketching enables virtual sketches and physical objects to "affect" each other through physical actuation and digital computation. In the existing AR sketching, the relationship between virtual and physical worlds is only one-directional -- while physical interaction can affect virtual sketches, virtual sketches have no return effect on the physical objects or environment. In contrast, bi-directional sketching interaction allows the seamless coupling between sketches and actuated TUIs. In this paper, we employ tabletop-size small robots (Sony Toio) and an iPad-based AR sketching tool to demonstrate the concept. In our system, virtual sketches drawn and simulated on an iPad (e.g., lines, walls, pendulums, and springs) can move, actuate, collide, and constrain physical Toio robots, as if virtual sketches and the physical objects exist in the same space through seamless coupling between AR and robot motion. This paper contributes a set of novel interactions and a design space of bi-directional AR sketching. We demonstrate a series of potential applications, such as tangible physics education, explorable mechanism, tangible gaming for children, and in-situ robot programming via sketching.

Created video streaming and network acceleration algorithms that have found their way in screens both large and small. It was also the first patent I had granted with Samsung, it was an equally challenging and fulfilling development crunch. Developed a mixed-reality tool for connected classrooms. It was a project with Samsung's Advanced Solutions Lab where one day I'm prototyping hardware for its Tangible User Interface, and another day I'm coding pattern recognition. While Harvard Business Review's D.J. Patil and Thomas Davenport declared Data Scientist "The Sexiest Job of the 21st Century", I have been fortunate enough to personally have been in a position to lead companies in various industries which were going through their own big data and data science transformations.

We think that ambient intelligence can expand playing house is one of the most popular play for young children. Therefore, we developed a prototype of cooking application for iPad. And we adopted Tangiblock invented by Bennesse Corporation as natural user interface. For cooking, users tap and rub a screen of the iPad with the ingredient and tool blocks. After that, a dish will be completed and the kind of dish decided automatically by combination of blocks. To make the user imagine eating the dish, we used an appetizing picture to express the dish. And we provided a recipe of the dish too, if they want to make it for real. As a result of observation, the game seemed able to attract children's interest in food, and we have think ambient intelligence has the potential to contribute to food education.